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ABSTRACT 

In 1968 the New York City Board of Education 
initiated a large scale test of a computer assisted instruction (CAI) 
program for drill and practice in elementary arithmetic. The program 
was a modified version of one developed by Dr. Patrick Suppes of 
Stanford University. An RCA Spectra 70/45 computer and 192 terminals 
located in elementary schools throughout the city were used. This , , 

report describes the methods used to evaluate the effectiveness of 
the CAI drill at each grade level, to determine the effect of the CAI 
drill on the attitudes of teachers, school administrators, parents ^ 

and students toward CAI, and to examine the effect of the CAI program 
on teaching procedures in elementary arithmetic. The results of 
various tests and observations are presented and interpreted. Based 
on the data collected in the study, some conclusions and 
recommendations are offered. (JY) 
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AN EVALUATION OF THE 196S- 1969 NEW YORK CITY 
COMPUTER ASSISTED INSTRUCTION PROJECT 
IN ELEMENTARY ARITHMETIC 



By 

Max Weiner, Samuel Malkin, Carl Helm and John Howell 



Introduction 

In September 1968 the New York City Board of Education initiated a project 
for a large-scale test-demonstration of a Computer Assisted Instruction (CAI) 
program for drill and practice in elementary arithmetic. This program was a 
modified version of an earlier arithmetic drill and practice program based on 
work done at Stanford University by Dr. Patrick Suppes and his staff, which had 
been field tested in various parts of the country. The New York 1968-’ 69 
experiment represented the first demonstration of a CAI arithmetic program of 
this type in a great metropolitan, center. The present report will describe some 
outcomes of the experiment in the first year of a proposed three-year test period. 
The "Hardware * ' 

The New York City CAI Project consists of an RCA Spectra 70/45 computer 
located at the project headquarters in Manhattan, four line concentrators at 
convenient locations in The Bronx, Manhattan, and Brooklyn, and 192 terminals 
located in elementary schools in those boroughs. 

The "Software" 



The CAI program for drill and practice in arithmetic provides exercises suit- 
able for use in grades two through six. It is designed to complement and support 
the instruction provided by the teacher, and hence it is intended neither to present 



concepts nor to provide explanations. It simply offers a carefully organized, varied, 
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and minutely gradea set of exercises, the exact selection and sequence of which 
is not rigidly predetermined. 

The CAI lessons are so devised that pupils generally can complete them in 
from two to ten minutes per lesson, whether the purpose is testing, drill only, 
or drill and review. The pupil receives his instructions and exercises and enters 
his responses at a terminal, which is to him "the computer'’ because it displays 
to him words and number t ombinations that are responsive to his own behavior. 

Plan of Evaluation 

The purpose of the evaluation of the 1968-69 New York City CAI Project was 
to describe the outcomes of the experiment in terms of its effects upon pupils and 
teachers. More particularly, its purpose was to seek to answer the following 
questions : 

1. What is the effect of the CAI drill and practice program on pupil 
achievement in arithmetic at each grade level, from grades 2 through 
grade 6? 

2. What is the effect of the CAI drill and practice program on pupil opinions 
and attitudes toward CAI, toward arithmetic, and toward learning in general? 
What is the effect of the CAI drill and practice program, and of the ad- 
justments necessary for its use, on teaching procedures in elementary 
arithmetic? 

4. What is the effect of the CAI drill and practice program on the opinions 
and attitudes of teachers, school administrators, and parents? 

In addition, information was sought on a number of subsidiary questions, which 
will be detailed further at appropriate points in the present report. 



The plan of evaluation was not - and, under the circumstances, could not 
have been - designed as a precise scientific experiment in which the variable 
under investigation was applied in a predetermined manner and all other contam- 
inating influences rigorously excluded. With respect to the question about pupil 
achievement in arithmetic, however, it was cast in the form of a quasi -experiment- 
tnat is, pupils receiving arithmetic instruction under CAI and non-CAI conditions, 
selected so as to be as comparable as possible beforehand, were compared on tests 
taken at the beginning and end of the school year. 

The Evaluation Team 

The City University of New York, by agreement with the New York City Board 
of Education, supplied the evaluation team, consisting of directors, consultants, 
and staff assistants. In addition, six profes -•rs in mathematics education, mem- 
bers of the faculties of the senior colleges of the City University, were employed 
to observe pupils in the CAI and non-CAI schools under study as part of the evalua- 
tion. They visited classrooms and CAI c<= :ers in the schools and also interviewed 
teachers, paraprofessionals, and school principals. 

Selection of Schools and Classe s 

Participation in the CAI experiment was accomplished by the staff of the 
Board of Education prior to the employment of the Evaluation team. The idea was 
first presented to district superintendents. Three expressed interest in applying 
the program in the schools under their jurisdiction. Each district superintendent, 
in his own fashion, then selected the schools that he judged most suitable, general 
after consulting with the school principals under his supervision. Later, in order 
to provide for comparison between CAI and non-CAI learning situations In which all 
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other conditions would be as nearly alike as possible, the district superintendents 
were asked to designate control schools on the basis of similarity to the CAT schools 
in the ethnic and socio-economic composition of their pupil populations and the 
judged equivalence of their administrative and instruct! onal staffs. 

Sixteen CAI schools and four non-CAI schools were designated. Arrangements 
were then made for the collection of data by achievement testing, observation, and 
interviews or questionnaires, in each school. 

Within schools the selection of classes to participate in the CAI experiment 
was left entirely to the principals with the stipulation that, intact class groups 
from each grade level from two to six should be included. As a rule, principals 
selected those teachers whom they judged capable of using the CAI program effect- 
ively and who expressed a desire or willingness to participate. 

Physical Arrangement of CAI Terminals in Schools 

On the average twelve CAI terminals per school were provided, the exact number 
depending on the particular circumstances in each school. The most usual arrange- 
ment for the use of the CAI equipment was to place all the terminals in a single 
room and bring the pupils there in groups. Under this plan a schedule for the use 
of the "CAI center” was worked out, and the paraprofessionals were assigned to es- 
cort groups of pupils to and from their classrooms as well as to supervise the use 
of the terminals at the center. In some schools, however, the CAI terminals were 
located in separate classrooms — two schools had single terminals in each of twelve 
and thirteen classrooms, respectively, and two others had two terminals in each 
of six classrooms and a single terminal in a seventh. Under these circumstances 
individual children worked by turns at the terminal while the teacher and the other 
children were carrying on othe activities. 

Spec In 1. Train in g Prog ram^ for T ouchers 1 Fr. man ro less -• o na 1 s 

Under the general plan of the New 'fork City CAI Project, as developed by tl 
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Board of Education, RCA provided for the initial training of participating teachers. 
This consisted of a 10-hour course held in the spring of 1968, which \?as attended 
by teachers from schools that would be using CAI in 1968-69. Although the teachers' 
strike in the fall of 1968 resulted in seme attrition, nearly 85 per cent of the 
original teachers were still in the program after the strike. Copies of the RCA 
Teachers’ Guid e, which was gone over in detail during initial training, were given 
each teacher for use during the school year . 

To assist the teachers in using CAI effectively, and especially to assist them 
in coordinating the CAI drill and practice program with the regular arithmetic cur- 
riculum, additional training and supervision were provided by the project director 
and by the curriculum specialists on his staff. At first district-wide workshops 
were organized, but after December it was decided that workshops should be conducted 
in each school so that problems peculiar to individual schools could be considered 
separately. 

The paraprofessionals were given six hours of training in the spring and two 
additional hours in the fall. Each paraprofessional received a manual covering 
some of the im, ertant aspects of the job: care of the CAI center (or of terminals 

in se; irate classrooms), maintaining discipline, changing paper and ribbons on the 
terminals, and a step-by-step procedure for detecting the sources of mechanical 
problems . 



Instruments 



Achievement Tests 

The Metropolitan Achievement Test (MAT) was chosen, after consultation with 
the Bureau of Research, because it is used extensively by the KcW Yor* City Board 
of Education in evaluating pupil achievement and lienee would provide comparability 
of measurement between the present study and other studies. The Levels, specific 

tests, and forms 

o 
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riimary II Battery (Form C) . Test 5, Arithmetic 
Part A: Concepts & Problem Solving. Test 5, 

Arithmetic Part B: Computation. 



Grades 3 and 4 



Elementary Arithmetic Test (Form C) . Test 1, 
Arithmetic Computation. Test 2, Arithmeti.c 
Problem Solving and Concepts. 



Grades 5 and 6 



Intermediate Arithmetic Test (Form CM). Test 1, 
Arithmetic Computation. Test 2, Arithmetic Problem 
Solving and Concepts. (Form CM ca.i be scored 
either by hand or by machine.) 



The same test forms were used as pre- and post-tests, and for analysis the raw 
scores were used. Thus no differences of scale affect the before-and after com- 



but summation across levels is impermiss? ole because the raw score scales of the 
three levels of the test are different. The experiment is sufficiently large, 
to permit all comparisons to be hell within school grades. The reader should keep 
in mind, in looking at the results that each level of the test is more difficult 
r’tu.n the preceding one. 

Observat i on Record 

Observation was considered essential to the evaluation despite its inherent 
difficulties, which wer^ intensified, as it turned out, by events affecting the 
whole city and its scnool system. An observation record form had been developed 
in advance, but the process of modifying it to make it acceptable to all persons 
concerned was interrupted by the teachers’ strike. After substantial delay and 
some necessary compromise, a revised version was developed and approved for use in 
the schools participating in the study. 

In teryicw Sche dules (Questionnaires ) 

To provide adequately for the reporting of the opinions and attitudes of the 



parisons within a level and lo°s of precision by conversion of scale is avoided, 
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kind of direct expression from participants. Therefore pupils, teachers, para- 
professionals; school administrators and parents were asked to respond to questions 
designed to elicit in a systematic way their perceptions of the CAI program and 
its effects. 
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TEST RESULTS 



Introduction 



Before <4i.S.C*ISSj.n& any of the test results, it must be err.ph .sized 
that the constraints involved in the collection o c the test data do not 
permit our saying with any decree of surety that any of the ".a ins found 
were attributable to CAI alone. There were many other variables which 
might have influenced the results. 

When all of the students' scores were examined , the mean rsw score 
gains between the January pre-test: and June post- test were higher, in 
all grades, for the CAI groups. This was true regardless of whether the 
data were grouped by total schools or by sex. 

The differences in gain scores were significant for grades 2, 3 and 
5 for all students. 

The MAT Results of Schools With a Predominantly Black or Puerto Ri can 
Popn.la tion 

Among the 20 schools which took part in the evaluation, there were 
5 CAI and 2 non- CAI schools with a 9074 or more Black or Puerto Rican 
population. Generally, these schools may be characterized as being 
in the poorest sections of the boroughs in which tnr y lo 05 ted . j. n era 

were significant differences in gains in grades 2 a no 2 in these scnools. 
When grouped by sex the CAI girls showed a significantly higher gain in 
grades 2 and 3. None of the differences were- significant for the boys. 
Additional Analyses 

It seemed appropriate to determine what the gains would be when the 
students in the non- CAI group were matched with the CAI group according 
to pre-test computation scores. This was dene and a total of 333 no tehee 
pairs were identified. Also n covariance analysis was undertaken. 
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All scores on the computation pre-test were listed in a re ending order 
by sen and credo in the data file. For each non-CAI computation pre-test 
j»av score a match was^ found by computer in the CAI group. If an exact 
match were not found, the program was set up to selcca a score witnin 1 
point of the score to be matched. \Tnere more than one CAI student earned 
a score which could be matched with a non-CAI student’s score, the computer 
selected one at random. 

As previously, the CAI students scored higher gains than the non-CAI 
students. This was also true when the data were listed by sex. The differ- 
ences were significant in all groupings for grades 2, 3 and 5. 

Data from Student Records Maintained by Computer 

CAI offers the educator the unique opportunity to study the process 
students go through in solving an arithmetic problem. In the present 
study an attempt was made to determine 1) how long students take to solve 
a problem (latency), 2) how many errors they make and 3) how often students 
were "timed out” by the computer, i.e. how often they failed to respond 
within the time limit (ten seconds) built into the CAT. program. Unfortunately 
the program was not designed to elicit this important information easily and 
economically . 

For purposes of comparison, the group was divided into two parts. 

The question which this division was intended to answer was whether 
the program was differentiated enough for students who are highest ard 
lowest in arithmetic ability. The selection of students, therefore, was 

^The covariance analysis confirmed the significant differences in 
grades 2, 3 and 5 for all the students in all the schools and grades 
2 and 3 for students in schools with a predominantly Black or 
Puerto Rican population. 
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on the basis of the pre-test total scores, i.e. scores from the compu- 
tation and concepts subtests combined. The results showed that at each 
grade level and for • eaPch problem answered, the low group required more 
time than the high group, 2) the mean error rate of the low group exceeded 
that of the high group, and 3) the mean number of time outs was also 
higher for the low group. These data are ba‘>ed on a very small sample 
of 66 students. 

Summary and Conclusions on Test Data 

A consistent pattern of gains was found for most CAI groups. That 
is, the CAI students earned higher gains in most grades, with signifi- 
cant differences found in the following: 

1. Grades 2, 3, and 5 for all students in all schools. 

2. Grades 2 and 3 for girls in all schools. 

3. Grade 0 2, 3, and 5 for boys in all schools. 

4. Grades 2 and 3 for all students in schools with predominantly 

Blach or Puerto Pdcan population. 

5. Grades 2 and 3 for girls in schools with a predc .iaantly Blach 
or Puerto Rican population. 

Classroom Activities - Observe?:- 0 , Reports 

1. The observers found no apprecia >le differences in instruction 1 '! 
procedures in classes usir* . C/:I as compared to those not using CAI. 

2. Over a period of several months no significant changes in teaching 
style, instructional procedures, or pupil behavior were noted in 
classes using CAI. 

3. while the observers were present, teachers tended to teach review 
lessons or lessons unrelated to the concepts for which th CAI program 
would provide cl ‘.11 and practice. 
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4. Mathematics lessons ran 3 ed in length from 20 to 60 minutes; most were 
approximately 30-35 minutes. 

5. With feu) m <?tafele exceptions, there was little evidence of grouping 
for mathematics instruction, or of other ways of providing for 
individual differences. 

6. The observers found no appreciable reduction in the amount of class 
time spent in drill and practice in GAI classes. In fact, many of 
the lessons observed, both CAI and non-CAI, could be classified as 
"practice” lessons. Drill and practice made up a significant part 
of each lesson; there was little reason to suspect that a computer 
was being used for that purpose exclusively. The percent of the 
lesson devoted to drill and practice ranged from 20% to 100% in all 
the classes, most teachers devoting about two- thirds of the lesson 
to drill and practice. 

7. A few of the teachers observed, devoted complete lessons preparing 
children to practice a particular topic on the computer. Furthermore, 
some teachers used a complete lesson for reteaching a topic because 
students had encountered unfamiliar symbols or abbreviations at the 
terminal. 

3. The instructional materials used in most classrooms included: 

textbooks, workbooks, rexographed materials, overhead transparencies, 
teacher demonstration models. Xn some classes, instructional materials 
were provided for each child, e.g,, fraction kits, clocks, cut-outs 
for sets, Cuisenaira rods, etc. 

9. Several teachers substituted computer practice for homework assignments. 

The Teacher and His Reaction to GAI 

1. Most teachers observed had an adequate understanding of the concepts 
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being taught. The exceptions to this ranged from several ueacne.s 
wh© were exceptionally competent in mathematics to those (not few 
in number ? whogfc background in mathematical concepts was uncertain 
or exceedingly sketchy, particularly in “modern math." 

2. Teachers used either textbooks or curriculum bulletins to plan 
their lessons. Few teachers indicated that they used the CAI 
teachers* guide in preparing lessons to any great 

3. Although teachers had reservations about the values of the CAI 
project, they felt that pupil enthusiasm and participation justified 
it and recommended its continuation. Teachers comments on CAI varied. 
Some positive comments concerning CAI were: 

a. it motivates good behavior; 

b. provides supplemental individualized drill, 

c. relieves teachers of some suilt feelings by providing children 
with needed drill that the teacher has no time to give; 

d. develops pupils’ speed in computation; 

e. has potential for keeping track of where each pupil xs. 

Some negative comments were: 



a. 



b. 



the lack of correlation of CAI and regular curriculum was a raj or 
s tumbling block; there was a growing gap between the classroom 
and CAI center activities; 

there was a need for greater control by the teacher of the program 



as a teaching supplement; 

c. some difficulties, such as discipline problems and classroom 
disruptions, arose as a resu. 
to terminals or CAI center; 



ult of children moving back and forth 



W 
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d. there was too little opportunity for teachers to discuss needs 
•nd; pgotilWRT with paraprofessionals, CAI coordinator, and 
coordinator. 

4. Some teachers believed that their own class tests and practice 
\ 

sheets were superior to and more relevant than what the cnildren 
practiced at the terminals. 

Students and Their Reaction to CAI 

X. Almost all teachers agreed that children enjoyed working on the 
computer, were enthusiastic about it, and were highly motivated 
to do well. This was particularly true of children of average 
ability. 

2. In spite of some frustrations and difficulties encountered by some 
children at the outset, most children have remained extremely 
enthusiastic about the "machine." 

3. Children of average and above average ability tended to enjoy the 
privacy of working at the terminal and even talked to the "machine." 

4. Teachers indicated that CAI encouraged pupils to commit their "number 
facts" to memory so that they would be able to respond in the time 
allowed by the computer. 

5. Some slower children tended to do a great deal of guessing and appeared 
to be playing games rather than practicing the skills needed to do their 
classwork. 

Pa r a p r o f e s s 1 ona 1 s 

X, The paraprofessionals were an extremely enthusiastic, interested and 

dedicated group. They took cheir role in this experiment most seriously 
and were anxious for the program to succeed and to continue. 
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2. Paraprofessionals felt that teachers needed to be more directly and 
actively ifWdWfid in the CAI activities and achievements of pupils. 

Many indicated a need for closer coordination of classwork and CAI 
work. Pupils who were either far ahead or far behind a class presented 
problems v?ith which they were unable to cope. 

3. The paraprofessionals considered the CAI center the most popular 
room in the school. 

Summary 

From the observations it appeared that CAI had little effect on 
teaching style or instructional procedures, on grouping and individualization 
of mathematics instruction in the classroom, on the amount of time devoted 
to drill and practice, and on increasing the emphasis on concept development 
in the classroom presentations. There was little coordination between CAI 
and the regular mathematics instruction — with CAI assuming a subordinate 
role. Thus, there were discrete classroom activities and discrete CAI 
activities either at a center or in a classroom. The prevaa.lr.ng attitude 
seemed to be that as long as the pupils enjoyed It, that was sufficient. 
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IKTER V IE17S AHb QIJ5ST T0 aKAIRES 

The reactions and attitudes of those connected with the CAI program 
(teachers/ students, school administrators, and parents) were obtained 
either by interview or by questionnaire. 

Administrators* Attitudes & Perceptions 

Principals were ashed a number of questions regarding pupils* work 
at the terminals. They overwhelmingly felt that (1) students enjoyed 
using the CAI terminals, (2) students enjoyed working problems at the 
terminals, and (3) students benefitted from working at the terminals. 

Teachers Estimate of Time 

Teachers were asked to estimate their preparation time and time 
spent in drill and practice. The responses indicated that CAI teachers 
said they spent a greater amount of time on preparation than did non- CAI 
teachers. As many as 8$ of CAI teachers spent as much as 75$ of their 
time in drill and practice. Teachers in both groups agreed that their 
drill & practice time would leave them enough time for teaching 
concepts. In CAI schools, 128 teachers agreed and 12 felt that the time 
was inadequate; and in non-CAI schools, 12 agreed, and one felt the time 
inadequate . 

Teachers Attitudes & Perceptions 




Teachers were asked if they were satisfied with their students 
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performance, in vi.C'.* of the time they devote to mathematics. In CAI 
Schools 65% o(~ th£ teachers indicated that they were satisfied, and 
27$ indicated that they were not satisfied. In non- CAI schools, 59$ 
indicated that they were satisfied, and 4-5 $ indicated that they were 
not. 

Parent Attidudes & Perceptions 

According to parents, fewer children in CAI schools show a strong 
dislike for arithmetic than do children in non-CAI schools. There is 
greater communication between the children and their parents in CAI 
schools than in non-CAI schools since only 4.7$ of the CAI parents . 
indicated that they were unaware of their children’s attitude toward 
arithmetic whereas 16$ of the non-CAI parents claimed they were unaware. 

CONCLUSIONS AM) HEC0r-S-ISM)ATI0N3 
Conclusions 

On the basis of the study of the New York City CAI project for the 
I968-I969 year of operation, the evaluation team has reached the 
following major conclusions: 

1. Very systematically, in nearly all groups, the CAI students 

made greater gains in arithmetic achievement as measured by the 
Metropolitan Achievement Tests (MAT) than the non-CAI students 
whom they were compared . In all cases in which the differences 
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were statistically significant, they favored ti CAI group. 
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Air>S wenng differences between CAT and non-CAI groups, each " 

grade was considered separately, and the following principal 

\ 

comparisons showed significant differences in favor of the 
CAI groups: 

a. Matched pairs of students drawn from entire student population: 
grades 2, 3 and 5. 

b. All students in all schools: grades 2, 3, and 5. 

c. All students in schools with a predominantly Black or Puerto 
Rican population (90$ or more) : grades 2 and 3. 

3 * Though the gains earned by many CAT groups were rather impressive, 
many variables not controlled in the present study could very well 
have contributed to the differences in the earned gains. Among 
these uncontrolled variables .are the following: 

a. selection of teachers. 

b. selection of schools. 

c. number of hours of arithmetic instruction. 

d. instruction for CAI children by personnel other than their 
teachers ( viz , paraprofessionals) and 

e. stress given to ar 5 .thmetic by faculty and students in CAI 
schools • 

4 . Observers were not able to detect differences in instructional 
procedures as between CAI and non-CAI classes. 

Observers found no appreciable reduction in the amount of time 
spent in drill and practice in CAI classes. Drill and practice 
usually made up a significant part of each lesson in CAI as w ell 
as non-CAI class. >. 
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Xi/ttle t)rrat teachers were using the CAT. Daily Status 

Reports and Concept Block Progress Reports for purposes of 
individualizing mathematics instruction, was encountered by the 
observers. The observers found that grouping for arithmetic 
instruction was infrequent in both the CAI and non-CAI groups. 

The observers concluded that a number of teachers needed further 
training in the subject matter of mathematics as well as in the 
appropriate use of CAI. 

A large percentage of all categories of respondents to interviews 
or questionnaires had a favorable attitude toward CAI and felt 
that work at the terminals helped the children learn arithmetic 
better. 

There was evidence, according to interviews and questionnaires, 
that CAI students tended to have a higher rate of communication 
with their parents about school work and arithmetic than did 
non-CAI students. 

CAI teachers reported having to spend a somewhat greater amount 
of time in lesson preparation. 

On the basis of very small samples, there was evidence found 
that the CAI arithmetic program (software) may not differentiate 
well enough for the high or low achievers. The software offered 
problc „ at five levels of difficulty based on an achievement 
pretest taken at the terminals. If this has provided sufficiently 
for individual differences one would assume that there would be 
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Very .Utfci e > If any, difference between error rates and time- 
outs for children at different ability levels. The findings 
showed that the low achievers require more time to do each 
problem and make more errors than the high achievers. 

Recommendations 

It seems quite important that future evaluations of student 
achievement in CAI take into account as many as possible the 
variables not controlled in the present study (see item 5, above). 
One area in particular, should be monitored, if not controlled, 
and that is the amount of instruction the children in both CAI 
and non-CAI schools receive in drill and practice. 

A more specific evaluation should be made of the software and 
its relevance for children at different levels of ability. It 
may well be, for example, that low achievers need more time at 
the termi n a l s while high achievers require less scheduled time. 

It may even be the case that the present software is inappropriate 
for one or both extreme groups. This recommendation was 
supported by observers in their reports as well as by the data 
from student records. 

There should be more CAI mathematics coordinators who are familiar 
with, the software available to the CAI teachers for assistance 
regarding content as well as method. 



